SherbyNet - /page/wikibook/entry.mu

Micron Document

<!DOCTYPE html>

<title>Byte-pair encoding</title>

</head>

<span id="openzim-page-title" class="mw-page-title-main"><span class="mw-page-title-main">Byte-pair encoding</span></span>

</h1>

</header>

<p>In <a href="Computing" title="Computing">computing</a>, <b>byte-pair encoding</b> (<b>BPE</b>),<sup id="cite_ref-CUsersJ_Gage_1994_1-0" class="reference"><a href="#cite_note-CUsersJ_Gage_1994-1"><span class="cite-bracket">[</span>1<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-2" class="reference"><a href="#cite_note-2"><span class="cite-bracket">[</span>2<span class="cite-bracket">]</span></a></sup> or <b>digram coding</b>,<sup id="cite_ref-3" class="reference"><a href="#cite_note-3"><span class="cite-bracket">[</span>3<span class="cite-bracket">]</span></a></sup> is an <a href="Algorithm" title="Algorithm">algorithm</a>, first described in 1994 by Philip Gage, for encoding strings of text into smaller strings by creating and using a translation table.<sup id="cite_ref-4" class="reference"><a href="#cite_note-4"><span class="cite-bracket">[</span>4<span class="cite-bracket">]</span></a></sup> A slightly modified version of the algorithm is used in <a href="Large_language_model" title="Large language model">large language model</a> tokenizers.

</p><p>The original version of the algorithm focused on compression. It replaces the highest-frequency pair of <a href="Byte" title="Byte">bytes</a> with a new byte that was not contained in the initial dataset. A <a href="Lookup_table" title="Lookup table">lookup table</a> of the replacements is required to rebuild the initial dataset. The modified version builds "tokens" (units of recognition) that match varying amounts of source text, from single characters (including single digits or single punctuation marks) to whole words (even long compound words).<sup id="cite_ref-5" class="reference"><a href="#cite_note-5"><span class="cite-bracket">[</span>5<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-6" class="reference"><a href="#cite_note-6"><span class="cite-bracket">[</span>6<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-7" class="reference"><a href="#cite_note-7"><span class="cite-bracket">[</span>7<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading2"><h2 id="Original_algorithm">Original algorithm</h2></div>

<p>The original BPE algorithm operates by iteratively replacing the most common contiguous sequences of characters in a target text with unused 'placeholder' bytes. The iteration ends when no sequences can be found, leaving the target text effectively compressed. Decompression can be performed by reversing this process, querying known placeholder terms against their corresponding denoted sequence, using a lookup table. In the original paper, this lookup table is encoded and stored alongside the compressed text.

</p>

<div class="mw-heading mw-heading3"><h3 id="Example">Example</h3></div>

<p>Suppose the data to be encoded is:<sup id="cite_ref-8" class="reference"><a href="#cite_note-8"><span class="cite-bracket">[</span>8<span class="cite-bracket">]</span></a></sup>

</p>

<pre>aaabdaaabac

</pre>

<p>The byte pair "aa" occurs most often, so it will be replaced by a byte that is not used in the data, such as "Z". Now there is the following data and replacement table:

</p>

<pre>ZabdZabac

Z=aa

</pre>

<p>Then the process is repeated with byte pair "ab", replacing it with "Y":

</p>

<pre>ZYdZYac

Y=ab

Z=aa

</pre>

<p>The only literal byte pair left occurs only once, and the encoding might stop here. Alternatively, the process could continue with <a href="Recursion" title="Recursion">recursive</a> byte-pair encoding, replacing "ZY" with "X":

</p>

<pre>XdXac

X=ZY

Y=ab

Z=aa

</pre>

<p>This data cannot be compressed further by byte-pair encoding because there are no pairs of bytes that occur more than once.

</p><p>To decompress the data, simply perform the replacements in the reverse order.

</p>

<div class="mw-heading mw-heading2"><h2 id="Modified_algorithm">Modified algorithm</h2></div>

<p>The original BPE algorithm is modified for use in <a href="Language_model" title="Language model">language modeling</a>, especially for large language models based on neural networks. Compared to the original BPE, the modified BPE does not aim to maximally compress text, but rather, to encode plaintext into "tokens", which are natural numbers.<sup id="cite_ref-9" class="reference"><a href="#cite_note-9"><span class="cite-bracket">[</span>9<span class="cite-bracket">]</span></a></sup> All the unique tokens found in a corpus are listed in a token vocabulary, the size of which, in the case of <a href="GPT-3.5" class="mw-redirect" title="GPT-3.5">GPT-3.5</a> and <a href="GPT-4" title="GPT-4">GPT-4</a>, is 100256.<sup id="cite_ref-10" class="reference"><a href="#cite_note-10"><span class="cite-bracket">[</span>10<span class="cite-bracket">]</span></a></sup>

</p><p>The modified tokenization algorithm initially treats the set of unique characters as 1-character-long <a href="N-grams" class="mw-redirect" title="N-grams">n-grams</a> (the initial tokens). Then, successively, the most frequent pair of adjacent tokens is merged into a new, longer n-gram and all instances of the pair are replaced by this new token. This is repeated until a vocabulary of prescribed size is obtained. Note that new words can always be constructed from final vocabulary tokens and initial-set characters.<sup id="cite_ref-2022Book_11-0" class="reference"><a href="#cite_note-2022Book_-11"><span class="cite-bracket">[</span>11<span class="cite-bracket">]</span></a></sup>

</p><p>This modified BPE approach has been extended from spoken language to sign language in recent years.<sup id="cite_ref-12" class="reference"><a href="#cite_note-12"><span class="cite-bracket">[</span>12<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Example_2">Example</h3></div>

<p>Suppose we are encoding the previous example of "aaabdaaabac", with a specified vocabulary size of 6, then it would first be encoded as "0, 0, 0, 1, 2, 0, 0, 0, 1, 0, 3" with a vocabulary of "a=0, b=1, d=2, c=3". Then it would proceed as before, and obtain "4, 5, 2, 4, 5, 0, 3" with a vocabulary of "a=0, b=1, d=2, c=3, aa=4, ab=5".

</p><p>So far this is essentially the same as before. However, if we only had specified a vocabulary size of 5, then the process would stop at vocabulary "a=0, b=1, d=2, c=3, aa=4", so that the example would be encoded as "4, 0, 1, 2, 4, 0, 1, 0, 3". Conversely, if we had specified a vocabulary size of 8, then it would be encoded as "7, 6, 0, 3", with a vocabulary of "a=0, b=1, d=2, c=3, aa=4, ab=5, aaab=6, aaabd=7". This is not maximally efficient, but the modified BPE does not aim to maximally compress a dataset, but aim to encode it efficiently for language model training.<sup id="cite_ref-13" class="reference"><a href="#cite_note-13"><span class="cite-bracket">[</span>13<span class="cite-bracket">]</span></a></sup>

</p>

<div class="mw-heading mw-heading3"><h3 id="Byte-level_BPE">Byte-level BPE</h3></div>

<p>In the above example, the output of the BPE is a vocabulary, which can be used to encode any text that is written with the letters "abcd". It will not be able to encode text containing other symbols, such as "no". Even giving each of the 26 letters an entry in the vocabulary, since there are many languages in the world using many different scripts, inevitably some symbols would be unencodable by such a vocabulary.

</p><p>One solution is to replace any unencodable symbol with a special symbol named UNK ("unknown").

</p><p>The byte-level BPE is another approach. It simply converts the text into <a href="UTF-8" title="UTF-8">UTF-8</a> first, and treat it as a stream of bytes. This guarantees that any text encoded in UTF-8 can be encoded by the BPE. This has been used in <a href="BERT_(language_model)" title="BERT (language model)">BERT</a>-like models like RoBERTa, BART, and DeBERTa, and <a href="Generative_pre-trained_transformer" title="Generative pre-trained transformer">GPT</a>-like models like <a href="GPT-2" title="GPT-2">GPT-2</a>.<sup id="cite_ref-14" class="reference"><a href="#cite_note-14"><span class="cite-bracket">[</span>14<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-15" class="reference"><a href="#cite_note-15"><span class="cite-bracket">[</span>15<span class="cite-bracket">]</span></a></sup><sup id="cite_ref-16" class="reference"><a href="#cite_note-16"><span class="cite-bracket">[</span>16<span class="cite-bracket">]</span></a></sup>

</p>

<li><a href="Sequitur_algorithm" title="Sequitur algorithm">Sequitur algorithm</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFGage1994" class="citation journal cs1">Gage, Philip (1994). <a rel="nofollow" class="external text" href="http://www.pennelynn.com/Documents/CUJ/HTML/94HTML/19940045.HTM">"A New Algorithm for Data Compression"</a>. <i>The C User Journal</i>.</cite></span>

</li>

<li id="cite_note-2"><span class="mw-cite-backlink"><b><a href="#cite_ref-2">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="http://www.drdobbs.com/article/print?articleId=184402829">"A New Algorithm for Data Compression"</a>. <i>Dr. Dobb's Journal</i>. 1 February 1994<span class="reference-accessdate">. Retrieved <span class="nowrap">10 August</span> 2020</span>.</cite></span>

</li>

<li id="cite_note-3"><span class="mw-cite-backlink"><b><a href="#cite_ref-3">^</a></b></span> <span class="reference-text"><cite id="CITEREFWittenMoffatBell1994" class="citation book cs1">Witten, Ian H.; Moffat, Alistair; Bell, Timothy C. (1994). <i>Managing Gigabytes</i>. New York: Van Nostrand Reinhold. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-442-01863-4</bdi>.</cite></span>

</li>

<li id="cite_note-4"><span class="mw-cite-backlink"><b><a href="#cite_ref-4">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://web.archive.org/web/20160326130908/http://www.csse.monash.edu.au/cluster/RJK/Compress/problem.html">"Byte Pair Encoding"</a>. Archived from <a rel="nofollow" class="external text" href="http://www.csse.monash.edu.au/cluster/RJK/Compress/problem.html">the original</a> on 2016-03-26.</cite></span>

</li>

<li id="cite_note-5"><span class="mw-cite-backlink"><b><a href="#cite_ref-5">^</a></b></span> <span class="reference-text"><cite id="CITEREFSennrichBirchHaddow2015" class="citation arxiv cs1">Sennrich, Rico; Birch, Alexandra; Haddow, Barry (2015-08-31). "Neural Machine Translation of Rare Words with Subword Units". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/1508.07909">1508.07909</a></span> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.CL">cs.CL</a>].</cite></span>

</li>

<li id="cite_note-6"><span class="mw-cite-backlink"><b><a href="#cite_ref-6">^</a></b></span> <span class="reference-text"><cite id="CITEREFBrownMannRyde_rSubbiah2020" class="citation arxiv cs1">Brown, Tom B.; Mann, Benjamin; Ryde r, Nick; Subbiah, Melanie; Kaplan, Jared; Dhariwal, Prafulla; Neelakantan, Arvind; Shyam, Pranav; Sastry, Girish; Askell, Amanda; Agarwal, Sandhini (2020-06-04). "Language Models are Few-Shot Learners". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/2005.14165">2005.14165</a></span> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.CL">cs.CL</a>].</cite></span>

</li>

<li id="cite_note-7"><span class="mw-cite-backlink"><b><a href="#cite_ref-7">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://github.com/google/sentencepiece">"google/sentencepiece"</a>. Google. 2021-03-02<span class="reference-accessdate">. Retrieved <span class="nowrap">2021-03-02</span></span>.</cite></span>

</li>

<li id="cite_note-8"><span class="mw-cite-backlink"><b><a href="#cite_ref-8">^</a></b></span> <span class="reference-text"><cite id="CITEREFCampesato2024" class="citation book cs1">Campesato, Oswald (2024-12-26). <i>Large Language Models for Developers: A Prompt-based Exploration of LLMs</i>. <a href="Walter_de_Gruyter_GmbH" class="mw-redirect" title="Walter de Gruyter GmbH">Walter de Gruyter GmbH</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-5015-2095-2</bdi>.</cite></span>

</li>

<li id="cite_note-9"><span class="mw-cite-backlink"><b><a href="#cite_ref-9">^</a></b></span> <span class="reference-text"><cite id="CITEREFZhangCaoYou2024" class="citation arxiv cs1">Zhang, Xiang; Cao, Juntai; You, Chenyu (2024). "Counting Ability of Large Language Models and Impact of Tokenization". <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/2410.19730">2410.19730</a></span> [<a rel="nofollow" class="external text" href="https://arxiv.org/archive/cs.CL">cs.CL</a>].</cite></span>

</li>

<li id="cite_note-10"><span class="mw-cite-backlink"><b><a href="#cite_ref-10">^</a></b></span> <span class="reference-text"><cite id="CITEREFRaschka2025" class="citation web cs1">Raschka, Sebastian (2025-01-17). <a rel="nofollow" class="external text" href="https://sebastianraschka.com/blog/2025/bpe-from-scratch.html">"Implementing A Byte Pair Encoding (BPE) Tokenizer From Scratch"</a>. <i>Sebastian Raschka, PhD</i><span class="reference-accessdate">. Retrieved <span class="nowrap">2025-07-05</span></span>.</cite></span>

</li>

<li id="cite_note-2022Book_-11"><span class="mw-cite-backlink"><b><a href="#cite_ref-2022Book_11-0">^</a></b></span> <span class="reference-text"><cite id="CITEREFPaaßGiesselbach2022" class="citation book cs1">Paaß, Gerhard; Giesselbach, Sven (2022). "Pre-trained Language Models". <i>Foundation Models for Natural Language Processing</i>. Artificial Intelligence: Foundations, Theory, and Algorithms. pp. <span class="nowrap">19–</span>78. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2F978-3-031-23190-2_2">10.1007/978-3-031-23190-2_2</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>9783031231902</bdi>.</cite></span>

</li>

<li id="cite_note-12"><span class="mw-cite-backlink"><b><a href="#cite_ref-12">^</a></b></span> <span class="reference-text"><cite id="CITEREFTaro_Miyazaki,_Sihan_Tan,_Tsubasa_Uchida,_Hiroyuki_Kaneko2024" class="citation journal cs1">Taro Miyazaki, Sihan Tan, Tsubasa Uchida, Hiroyuki Kaneko (May 25, 2024). <a rel="nofollow" class="external text" href="https://aclanthology.org/2024.signlang-1.29.pdf">"Sign Language Translation with Gloss Pair Encoding"</a> <span class="cs1-format">(PDF)</span>. <i>Proceedings of the 11th Workshop on the Representation and Processing of Sign Languages</i>.</cite><span class="cs1-maint citation-comment"><code class="cs1-code">{{cite journal}}</code>: CS1 maint: multiple names: authors list (link)</span></span>

</li>

<li id="cite_note-13"><span class="mw-cite-backlink"><b><a href="#cite_ref-13">^</a></b></span> <span class="reference-text"><cite id="CITEREFPai2025" class="citation book cs1">Pai, Suhas (2025-03-06). <i>Designing Large Language Model Applications: A Holistic Approach to LLMs</i>. <a href="O'Reilly_Media" title="O'Reilly Media">O'Reilly Media</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-0981-5046-4</bdi>.</cite></span>

</li>

<li id="cite_note-14"><span class="mw-cite-backlink"><b><a href="#cite_ref-14">^</a></b></span> <span class="reference-text"><cite class="citation web cs1"><a rel="nofollow" class="external text" href="https://huggingface.co/learn/nlp-course/en/chapter6/5">"Byte-Pair Encoding tokenization"</a>. <i>Hugging Face NLP Course</i><span class="reference-accessdate">. Retrieved <span class="nowrap">2025-01-27</span></span>.</cite></span>

</li>

<li id="cite_note-15"><span class="mw-cite-backlink"><b><a href="#cite_ref-15">^</a></b></span> <span class="reference-text"><cite id="CITEREFYıldırımChenaghlu2021" class="citation book cs1">Yıldırım, Savaş; Chenaghlu, Meysam Asgari (2021-09-15). <i>Mastering Transformers: Build state-of-the-art models from scratch with advanced natural language processing techniques</i>. <a href="Packt_Publishing_Ltd" class="mw-redirect" title="Packt Publishing Ltd">Packt Publishing Ltd</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-1-80107-889-4</bdi>.</cite></span>

</li>

<li id="cite_note-16"><span class="mw-cite-backlink"><b><a href="#cite_ref-16">^</a></b></span> <span class="reference-text"><cite id="CITEREFWangCho2020" class="citation journal cs1">Wang, Changhan; Cho, Kyunghyun (2020-04-03). <a rel="nofollow" class="external text" href="https://doi.org/10.1609%2Faaai.v34i05.6451">"Neural Machine Translation with Byte-Level Subwords"</a>. <i>Proceedings of the AAAI Conference on Artificial Intelligence</i>. <b>34</b> (5): <a rel="nofollow" class="external text" href="https://pdfguru.com/compress-pdf">9154–9160</a>. <a href="ArXiv_(identifier)" class="mw-redirect" title="ArXiv (identifier)">arXiv</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://arxiv.org/abs/1909.03341">1909.03341</a></span>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<span class="id-lock-free" title="Freely accessible"><a rel="nofollow" class="external text" href="https://doi.org/10.1609%2Faaai.v34i05.6451">10.1609/aaai.v34i05.6451</a></span>. <a href="ISSN_(identifier)" class="mw-redirect" title="ISSN (identifier)">ISSN</a> <a rel="nofollow" class="external text" href="https://search.worldcat.org/issn/2374-3468">2374-3468</a>.</cite></span>

</li>

</ol></div></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hlist dl,.mw-parser-output .hlist ol,.mw-parser-output .hlist ul{margin:0;padding:0}.mw-parser-output .hlist dd,.mw-parser-output .hlist dt,.mw-parser-output .hlist li{margin:0;display:inline}.mw-parser-output .hlist.inline,.mw-parser-output .hlist.inline dl,.mw-parser-output .hlist.inline ol,.mw-parser-output .hlist.inline ul,.mw-parser-output .hlist dl dl,.mw-parser-output .hlist dl ol,.mw-parser-output .hlist dl ul,.mw-parser-output .hlist ol dl,.mw-parser-output .hlist ol ol,.mw-parser-output .hlist ol ul,.mw-parser-output .hlist ul dl,.mw-parser-output .hlist ul ol,.mw-parser-output .hlist ul ul{display:inline}.mw-parser-output .hlist .mw-empty-li{display:none}.mw-parser-output .hlist dt::after{content:": "}.mw-parser-output .hlist dd::after,.mw-parser-output .hlist li::after{content:" · ";font-weight:bold}.mw-parser-output .hlist dd:last-child::after,.mw-parser-output .hlist dt:last-child::after,.mw-parser-output .hlist li:last-child::after{content:none}.mw-parser-output .hlist dd dd:first-child::before,.mw-parser-output .hlist dd dt:first-child::before,.mw-parser-output .hlist dd li:first-child::before,.mw-parser-output .hlist dt dd:first-child::before,.mw-parser-output .hlist dt dt:first-child::before,.mw-parser-output .hlist dt li:first-child::before,.mw-parser-output .hlist li dd:first-child::before,.mw-parser-output .hlist li dt:first-child::before,.mw-parser-output .hlist li li:first-child::before{content:" (";font-weight:normal}.mw-parser-output .hlist dd dd:last-child::after,.mw-parser-output .hlist dd dt:last-child::after,.mw-parser-output .hlist dd li:last-child::after,.mw-parser-output .hlist dt dd:last-child::after,.mw-parser-output .hlist dt dt:last-child::after,.mw-parser-output .hlist dt li:last-child::after,.mw-parser-output .hlist li dd:last-child::after,.mw-parser-output .hlist li dt:last-child::after,.mw-parser-output .hlist li li:last-child::after{content:")";font-weight:normal}.mw-parser-output .hlist ol{counter-reset:listitem}.mw-parser-output .hlist ol>li{counter-increment:listitem}.mw-parser-output .hlist ol>li::before{content:" "counter(listitem)"\a0 "}.mw-parser-output .hlist dd ol>li:first-child::before,.mw-parser-output .hlist dt ol>li:first-child::before,.mw-parser-output .hlist li ol>li:first-child::before{content:" ("counter(listitem)"\a0 "}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1236075235">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbox{box-sizing:border-box;border:1px solid #a2a9b1;width:100%;clear:both;font-size:88%;text-align:center;padding:1px;margin:1em auto 0}.mw-parser-output .navbox .navbox{margin-top:0}.mw-parser-output .navbox+.navbox,.mw-parser-output .navbox+.navbox-styles+.navbox{margin-top:-1px}.mw-parser-output .navbox-inner,.mw-parser-output .navbox-subgroup{width:100%}.mw-parser-output .navbox-group,.mw-parser-output .navbox-title,.mw-parser-output .navbox-abovebelow{padding:0.25em 1em;line-height:1.5em;text-align:center}.mw-parser-output .navbox-group{white-space:nowrap;text-align:right}.mw-parser-output .navbox,.mw-parser-output .navbox-subgroup{background-color:#fdfdfd}.mw-parser-output .navbox-list{line-height:1.5em;border-color:#fdfdfd}.mw-parser-output .navbox-list-with-group{text-align:left;border-left-width:2px;border-left-style:solid}.mw-parser-output tr+tr>.navbox-abovebelow,.mw-parser-output tr+tr>.navbox-group,.mw-parser-output tr+tr>.navbox-image,.mw-parser-output tr+tr>.navbox-list{border-top:2px solid #fdfdfd}.mw-parser-output .navbox-title{background-color:#ccf}.mw-parser-output .navbox-abovebelow,.mw-parser-output .navbox-group,.mw-parser-output .navbox-subgroup .navbox-title{background-color:#ddf}.mw-parser-output .navbox-subgroup .navbox-group,.mw-parser-output .navbox-subgroup .navbox-abovebelow{background-color:#e6e6ff}.mw-parser-output .navbox-even{background-color:#f7f7f7}.mw-parser-output .navbox-odd{background-color:transparent}.mw-parser-output .navbox .hlist td dl,.mw-parser-output .navbox .hlist td ol,.mw-parser-output .navbox .hlist td ul,.mw-parser-output .navbox td.hlist dl,.mw-parser-output .navbox td.hlist ol,.mw-parser-output .navbox td.hlist ul{padding:0.125em 0}.mw-parser-output .navbox .navbar{display:block;font-size:100%}.mw-parser-output .navbox-title .navbar{float:left;text-align:left;margin-right:0.5em}body.skin--responsive .mw-parser-output .navbox-image img{max-width:none!important}@media print{body.ns-0 .mw-parser-output .navbox{display:none!important}}

/* end https://en.wikipedia.org/ */

</style></div><div role="navigation" class="navbox" aria-labelledby="Data_compression_methods241" style="padding:3px"><table class="nowraplinks hlist mw-collapsible autocollapse navbox-inner" style="border-spacing:0;background:transparent;color:inherit"><tbody><tr><th scope="col" class="navbox-title" colspan="2"><style data-mw-deduplicate="TemplateStyles:r1239400231">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbar{display:inline;font-size:88%;font-weight:normal}.mw-parser-output .navbar-collapse{float:left;text-align:left}.mw-parser-output .navbar-boxtext{word-spacing:0}.mw-parser-output .navbar ul{display:inline-block;white-space:nowrap;line-height:inherit}.mw-parser-output .navbar-brackets::before{margin-right:-0.125em;content:"[ "}.mw-parser-output .navbar-brackets::after{margin-left:-0.125em;content:" ]"}.mw-parser-output .navbar li{word-spacing:-0.125em}.mw-parser-output .navbar a>span,.mw-parser-output .navbar a>abbr{text-decoration:inherit}.mw-parser-output .navbar-mini abbr{font-variant:small-caps;border-bottom:none;text-decoration:none;cursor:inherit}.mw-parser-output .navbar-ct-full{font-size:114%;margin:0 7em}.mw-parser-output .navbar-ct-mini{font-size:114%;margin:0 4em}html.skin-theme-clientpref-night .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}@media(prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}}@media print{.mw-parser-output .navbar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div id="Data_compression_methods241" style="font-size:114%;margin:0 4em"><a href="Data_compression" title="Data compression">Data compression</a> methods</div></th></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Lossless_compression" title="Lossless compression">Lossless</a><br>type</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Entropy_coding" title="Entropy coding">Entropy</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Adaptive_coding" title="Adaptive coding">Adaptive coding</a></li>

<li><a href="Arithmetic_coding" title="Arithmetic coding">Arithmetic</a></li>

<li><a href="Asymmetric_numeral_systems" title="Asymmetric numeral systems">Asymmetric numeral systems</a></li>

<li><a href="Golomb_coding" title="Golomb coding">Golomb</a></li>

<li><a href="Huffman_coding" title="Huffman coding">Huffman</a>

<ul><li><a href="Adaptive_Huffman_coding" title="Adaptive Huffman coding">Adaptive</a></li>

<li><a href="Canonical_Huffman_code" title="Canonical Huffman code">Canonical</a></li>

<li><a href="Modified_Huffman_coding" title="Modified Huffman coding">Modified</a></li></ul></li>

<li><a href="Range_coding" title="Range coding">Range</a></li>

<li><a href="Shannon_coding" title="Shannon coding">Shannon</a></li>

<li><a href="Shannon%E2%80%93Fano_coding" title="Shannon–Fano coding">Shannon–Fano</a></li>

<li><a href="Shannon%E2%80%93Fano%E2%80%93Elias_coding" title="Shannon–Fano–Elias coding">Shannon–Fano–Elias</a></li>

<li><a href="Tunstall_coding" title="Tunstall coding">Tunstall</a></li>

<li><a href="Unary_coding" title="Unary coding">Unary</a></li>

<li><a href="Universal_code_(data_compression)" title="Universal code (data compression)">Universal</a>

<ul><li><a href="Exponential-Golomb_coding" title="Exponential-Golomb coding">Exp-Golomb</a></li>

<li><a href="Fibonacci_coding" title="Fibonacci coding">Fibonacci</a></li>

<li><a href="Elias_gamma_coding" title="Elias gamma coding">Gamma</a></li>

<li><a href="Levenshtein_coding" title="Levenshtein coding">Levenshtein</a></li></ul></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Dictionary_coder" title="Dictionary coder">Dictionary</a></th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul>

<li><a href="LZ77_and_LZ78" title="LZ77 and LZ78">Lempel–Ziv</a>

<li><a href="Snappy_(compression)" title="Snappy (compression)">Snappy</a></li></ul></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Other</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<li><a href="Delta_encoding" title="Delta encoding">Delta</a>

<ul><li><a href="Incremental_encoding" title="Incremental encoding">Incremental</a></li></ul></li>

<li><a href="Grammar-based_code" title="Grammar-based code">Grammar</a>

<li><a href="Sequitur_algorithm" title="Sequitur algorithm">Sequitur</a></li></ul></li>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Hybrid</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li>LZ77 + Huffman

<ul><li><a href="Deflate" title="Deflate">Deflate</a></li>

<li>LZ77 + ANS

<ul><li><a href="LZFSE" title="LZFSE">LZFSE</a></li></ul></li>

<li>LZ77 + Huffman + ANS

<ul><li><a href="Zstd" title="Zstd">Zstandard</a></li></ul></li>

<li>LZ77 + Huffman + context

<ul><li><a href="Brotli" title="Brotli">Brotli</a></li></ul></li>

<li>LZSS + Huffman

<li>LZ77 + Range

<li>LZHAM</li></ul></li>

<li>RLE + BWT + MTF + Huffman

</div></td></tr></tbody></table><div></div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Lossy_compression" title="Lossy compression">Lossy</a><br>type</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Transform_coding" title="Transform coding">Transform</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Discrete_cosine_transform" title="Discrete cosine transform">Discrete cosine transform</a>

<li><a href="Wavelet_transform" title="Wavelet transform">Wavelet</a>

<ul><li><a href="Daubechies_wavelet" title="Daubechies wavelet">Daubechies</a></li>

<li><a href="Set_partitioning_in_hierarchical_trees" title="Set partitioning in hierarchical trees">SPIHT</a></li></ul></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Predictive</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Adaptive_differential_pulse-code_modulation" title="Adaptive differential pulse-code modulation">ADPCM</a></li></ul></li>

<ul><li><a href="Algebraic_code-excited_linear_prediction" title="Algebraic code-excited linear prediction">ACELP</a></li>

<li>Motion

<ul><li><a href="Motion_compensation" title="Motion compensation">Compensation</a></li>

<li><a href="Motion_estimation" title="Motion estimation">Estimation</a></li>

<li><a href="Motion_vector" class="mw-redirect" title="Motion vector">Vector</a></li></ul></li>

<li><a href="Psychoacoustics" title="Psychoacoustics">Psychoacoustic</a></li></ul>

</div></td></tr></tbody></table><div></div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Data_compression#Audio" title="Data compression">Audio</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%">Concepts</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<li><a href="Companding" title="Companding">Companding</a></li>

<li><a href="Convolution" title="Convolution">Convolution</a></li>

<li><a href="Dynamic_range" title="Dynamic range">Dynamic range</a></li>

<li><a href="Latency_(audio)" title="Latency (audio)">Latency</a></li>

<li><a href="Nyquist%E2%80%93Shannon_sampling_theorem" title="Nyquist–Shannon sampling theorem">Nyquist–Shannon theorem</a></li>

<li><a href="Sampling_(signal_processing)" title="Sampling (signal processing)">Sampling</a></li>

<li><a href="Silence_compression" title="Silence compression">Silence compression</a></li>

<li><a href="Sound_quality" title="Sound quality">Sound quality</a></li>

<li><a href="Speech_coding" title="Speech coding">Speech coding</a></li>

<li><a href="Sub-band_coding" title="Sub-band coding">Sub-band coding</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Audio_codec" title="Audio codec">Codec</a><br>parts</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Adaptive_differential_pulse-code_modulation" title="Adaptive differential pulse-code modulation">ADPCM</a></li>

<ul><li><a href="Algebraic_code-excited_linear_prediction" title="Algebraic code-excited linear prediction">ACELP</a></li>

<li><a href="Psychoacoustics" title="Psychoacoustics">Psychoacoustic model</a></li></ul>

</div></td></tr></tbody></table><div></div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Image_compression" title="Image compression">Image</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%">Concepts</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Chroma_subsampling" title="Chroma subsampling">Chroma subsampling</a></li>

<li><a href="Coding_tree_unit" title="Coding tree unit">Coding tree unit</a></li>

<li><a href="Color_space" title="Color space">Color space</a></li>

<li><a href="Compression_artifact" title="Compression artifact">Compression artifact</a></li>

<li><a href="Image_resolution" title="Image resolution">Image resolution</a></li>

<li><a href="Macroblock" title="Macroblock">Macroblock</a></li>

<li><a href="Pixel" title="Pixel">Pixel</a></li>

<li><a href="Quantization_(image_processing)" title="Quantization (image processing)">Quantization</a></li>

<li><a href="Standard_test_image" title="Standard test image">Standard test image</a></li>

<li><a href="Texture_compression" title="Texture compression">Texture compression</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Methods</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Chain_code" title="Chain code">Chain code</a></li>

<li><a href="Deflate" title="Deflate">Deflate</a></li>

<li><a href="Fractal_compression" title="Fractal compression">Fractal</a></li>

<li><a href="Wavelet_transform" title="Wavelet transform">Wavelet</a>

<ul><li><a href="Daubechies_wavelet" title="Daubechies wavelet">Daubechies</a></li>

<li><a href="Set_partitioning_in_hierarchical_trees" title="Set partitioning in hierarchical trees">SPIHT</a></li></ul></li></ul>

</div></td></tr></tbody></table><div></div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Data_compression#Video" title="Data compression">Video</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em"></div><table class="nowraplinks navbox-subgroup" style="border-spacing:0"><tbody><tr><th scope="row" class="navbox-group" style="width:1%">Concepts</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<li><a href="Display_resolution" title="Display resolution">Display resolution</a></li>

<li><a href="Film_frame" title="Film frame">Frame</a></li>

<li><a href="Frame_rate" title="Frame rate">Frame rate</a></li>

<li><a href="Video_compression_picture_types" title="Video compression picture types">Frame types</a></li>

<li><a href="Interlaced_video" title="Interlaced video">Interlace</a></li>

<li><a href="Video#Characteristics_of_video_streams" title="Video">Video characteristics</a></li>

<li><a href="Video_quality" title="Video quality">Video quality</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Video_codec" title="Video codec">Codec</a><br>parts</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<li><a href="Deblocking_filter" title="Deblocking filter">Deblocking filter</a></li>

<li><a href="Lapped_transform" title="Lapped transform">Lapped transform</a></li>

<li>Motion

<ul><li><a href="Motion_compensation" title="Motion compensation">Compensation</a></li>

<li><a href="Motion_estimation" title="Motion estimation">Estimation</a></li>

<li><a href="Motion_vector" class="mw-redirect" title="Motion vector">Vector</a></li></ul></li>

<li><a href="Wavelet_transform" title="Wavelet transform">Wavelet</a>

<ul><li><a href="Daubechies_wavelet" title="Daubechies wavelet">Daubechies</a></li>

</div></td></tr></tbody></table><div></div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%"><a href="Information_theory" title="Information theory">Theory</a></th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Compressed_data_structure" title="Compressed data structure">Compressed data structures</a>

<ul><li><a href="Compressed_suffix_array" title="Compressed suffix array">Compressed suffix array</a></li>

<li><a href="FM-index" title="FM-index">FM-index</a></li></ul></li>

<li><a href="Entropy_(information_theory)" title="Entropy (information theory)">Entropy</a></li>

<li><a href="Information_theory" title="Information theory">Information theory</a>

<ul><li><a href="Timeline_of_information_theory" title="Timeline of information theory">Timeline</a></li></ul></li>

<li><a href="Kolmogorov_complexity" title="Kolmogorov complexity">Kolmogorov complexity</a></li>

<li><a href="Prefix_code" title="Prefix code">Prefix code</a></li>

<li><a href="Quantization_(signal_processing)" title="Quantization (signal processing)">Quantization</a></li>

<li><a href="Rate%E2%80%93distortion_theory" title="Rate–distortion theory">Rate–distortion</a></li>

<li><a href="Redundancy_(information_theory)" title="Redundancy (information theory)">Redundancy</a></li>

<li><a href="Data_compression_symmetry" title="Data compression symmetry">Symmetry</a></li>

<li><a href="Smallest_grammar_problem" title="Smallest grammar problem">Smallest grammar problem</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">Community</th><td class="navbox-list-with-group navbox-list navbox-even" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Hutter_Prize" title="Hutter Prize">Hutter Prize</a></li></ul>

</div></td></tr><tr><th scope="row" class="navbox-group" style="width:1%">People</th><td class="navbox-list-with-group navbox-list navbox-odd" style="width:100%;padding:0"><div style="padding:0 0.25em">

<ul><li><a href="Mark_Adler" title="Mark Adler">Mark Adler</a></li>

</div></td></tr></tbody></table></div></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-08-04" href="https://en.wikipedia.org/wiki/?title=Byte-pair_encoding&oldid=1304243756">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>